Optimal Linear Combinations of Neural Networks 1

Neural network (NN)-based modeling often involves trying multiple networks with diierent architectures and training parameters in order to achieve acceptable model accuracy. Typically, one of the trained NNs is chosen as best, while the rest are discarded. Hashem and Schmeiser 25] proposed using optimal linear combinations of a number of trained neural networks instead of using a single best network. Combining the trained networks may help integrate the knowledge acquired by the component networks and thus improve model accuracy. In this paper, we discuss and extend the idea of optimal linear combinations (OLCs) of neural networks and derive closed-form expressions for four cases of OLCs. We present two algorithms for selecting the component networks for the combination to improve the generalization ability of OLCs. Our experimental results demonstrate signiicant improvements in model accuracy as a result of using OLCs, as compared to the best network.